Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add publish worker #115

Merged
merged 1 commit into from
Aug 27, 2024
Merged

Add publish worker #115

merged 1 commit into from
Aug 27, 2024

Conversation

richardhuaaa
Copy link
Contributor

@richardhuaaa richardhuaaa commented Aug 21, 2024

  • Adds a publish worker that performs in-order insertion into the gateway_envelopes table
  • Adds basic validation of the client envelope on publish
  • Store the topic on the staged_originated_envelopes table - we extract this during the API call, so that the publish worker doesn't need to do any additional unmarshaling or validation.

Would particularly love feedback on the error handling in the worker, and if there's any test cases I should add to service_test.go.

Copy link
Contributor Author

richardhuaaa commented Aug 21, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @richardhuaaa and the rest of your teammates on Graphite Graphite

@richardhuaaa richardhuaaa mentioned this pull request Aug 21, 2024
@richardhuaaa richardhuaaa force-pushed the 08-20-add_db_subscription branch from 647b702 to 92f5ac8 Compare August 21, 2024 04:36
@richardhuaaa richardhuaaa force-pushed the 08-20-add_publish_worker branch from b994784 to cb09162 Compare August 21, 2024 04:36
@richardhuaaa richardhuaaa changed the title add publish worker Add publish worker Aug 21, 2024
@richardhuaaa richardhuaaa force-pushed the 08-20-add_db_subscription branch from 92f5ac8 to 6ce44be Compare August 23, 2024 07:15
@richardhuaaa richardhuaaa force-pushed the 08-20-add_publish_worker branch 2 times, most recently from 177ea52 to 16b2940 Compare August 23, 2024 07:25
@richardhuaaa richardhuaaa force-pushed the 08-20-add_db_subscription branch from 6ce44be to a100969 Compare August 23, 2024 07:29
@richardhuaaa richardhuaaa force-pushed the 08-20-add_publish_worker branch from 16b2940 to d27f00a Compare August 23, 2024 07:29
@richardhuaaa richardhuaaa force-pushed the 08-20-add_db_subscription branch 2 times, most recently from 4467348 to db44f65 Compare August 26, 2024 05:36
@richardhuaaa richardhuaaa force-pushed the 08-20-add_publish_worker branch 3 times, most recently from 2b75efb to 0752a9b Compare August 26, 2024 18:16
@richardhuaaa richardhuaaa marked this pull request as ready for review August 26, 2024 18:20
pkg/api/publishWorker.go Outdated Show resolved Hide resolved
pkg/api/publishWorker.go Outdated Show resolved Hide resolved
)
if err != nil {
logger.Error("Failed to insert gateway envelope", zap.Error(err))
return false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a risk here that we infinitely retry on this step or one of the steps above in an unanticipated error case. I think that we should be okay here but a second pair of eyes can't hurt

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know why the proto marshaling would fail, but if it did we would be in an endless loop of retries here.

Would probably have to be a maliciously crafted payload

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my research, it seems like it fails if some expected fields of the proto are not set. In this case, the proto is OriginatorEnvelope with two fields that we are setting here, so I expect that we should be able to prevent this error happening.

I still think we should catch the error just in case though, firstly so that we can detect it, secondly so that we don't screw up payload ordering if there is an unanticipated error type that is super common.

Base automatically changed from 08-20-add_db_subscription to main August 26, 2024 22:56
@richardhuaaa richardhuaaa force-pushed the 08-20-add_publish_worker branch from 0752a9b to a8cef43 Compare August 26, 2024 23:18
@richardhuaaa richardhuaaa requested a review from neekolas August 27, 2024 00:36
@richardhuaaa richardhuaaa merged commit 650bf24 into main Aug 27, 2024
5 checks passed
@richardhuaaa richardhuaaa deleted the 08-20-add_publish_worker branch August 27, 2024 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants